Skip to content

Add Axum production startup and bounded shutdown - #406

Open
ChristianPavilonis wants to merge 16 commits into
issue-459-outbound-compatfrom
issue-392-396-production-hosting-spec
Open

ChristianPavilonis wants to merge 16 commits into
issue-459-outbound-compatfrom
issue-392-396-production-hosting-spec

Conversation

@ChristianPavilonis

@ChristianPavilonis ChristianPavilonis commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Current review readiness — 2026-11-10

Ready for review at the publication head below. All29 current hosted checks passed. Reviewers: aram356, prk-Jr, dhruv8sh.

Review transitions are independent of TS work and the combined integration audit; #462 remains draft. This current status supersedes historical draft-only prerequisites below without claiming unexecuted deployment/manual acceptance complete. All three reviewers are requested. Stack465 is the current grouping; #453 is excluded.

Current review-stack checkpoint

Publication head: 9f090e573206cf0d8c21c1ce0067afffd69f12db. This cleanup preserves the existing feature unit and PR identity. The additional change is the completed generated-native build correction: image-only code now follows the scaffold's existing Fastly-example selection, with no request-policy change. Its focused generator/native proof is retained; fresh per-branch hosted checks are not assumed green.

Review order: #275 → #406 → #407 → #409 → #414 → #422 → #439 → #454. Draft integration checkpoint: #462. New SDK/group/codec/phase development and unrelated TS work are not prerequisites or additions to this PR. Earlier notes below retain their original tested revisions and limits; they are not a whole-tree pass for this new head.

Earlier implementation and proof notes

RC refresh checkpoint, 2026-11-10

Current head: 1d8e42e1cba52610e6655965e9941f42680b53ab.

Current #275 hosting port, with bounded startup/config/ingress and existing callback/egress ownership retained. The invalid-present native-limit and Windows prefix-identity capture bugs found by review are fixed. Windows semantics were simulated at the private parser boundary, not run on Windows.

Fresh CI exposed missing fixture validator metadata, an always-compiled test listener helper and a trigger assertion still expecting main-only PRs. The fixture lock is synced, the helper now has the same test/test-utils gate as its caller, and the exact workflow test now requires all PR bases while retaining main-only push. No lint suppression or production/deployment trigger expansion. Parent proof includes 301 Axum unit tests, axum-only strict Clippy and 35 Node CI tests.

Draft pending fresh hosted checks and the combined RC audit. Two unchanged native default-parallel deadline misses remain recorded and unclassified; serial passes and untouched-foundation parallel passes do not clear them. No timing bound was relaxed. No deployment, main merge, security-alert dismissal or review-thread resolution occurred. Historical design/review context follows.


Summary

  • Add explicit production startup so required settings, stores, transport, and app-owned initialization succeed before accepting traffic. Keep permissive development entrypoints.
  • Extend the existing native HTTP/1 runner with SIGINT/Unix SIGTERM handling, tracked graceful draining, a finite shutdown budget, and forced non-success exit.
  • Add portable app-owned probes, shared typed startup config loading, retained handler state, and subprocess acceptance tests.

This PR is stacked on #275, with base branch docs/outbound-http-spec. The implementation started at 5527b58. It remains draft because #275 has advanced to 91f4af38, five parent-only commits beyond the common base 5527b58. Parent integration and re-audit, including the previously identified dev_server.rs conflict, remain outstanding. The recorded checks are for this branch, not integration with the current prerequisite. This relationship audit performed no merge or rebase. Integrate the current contracts and repeat affected validation before marking ready.

Changes

Crate / File Change
edgezero-adapter-axum/src/dev_server.rs, run_options.rs Explicit production and embedding APIs, prepared registries, app initializer, lifecycle ownership, signals, and bounded drain
Axum connection/service modules Private dispatch gate, live lifecycle reader, and graceful HTTP/1 connection shutdown
Axum config/KV stores Strict required config snapshots, explicit roots, same-name KV aliases, and redacted startup failures without delete-and-restart advice
edgezero-core/src/app.rs, router.rs Startup-only typed state insertion with copy-on-write isolation and old-admission invalidation
edgezero-core/src/extractor.rs, probe.rs Context-free typed config loading shared with request extraction, plus ordinary readiness/liveness actions
Native and portable test fixtures Real process signals, finite/forced drain, startup failure/redaction, embedding, restart persistence, aliasing, stopped-writer backup/restore, and projected read-only config
Production hosting design Contracts, acceptance boundaries, executed checks, and remaining integration/deployment work

Closes

Closes #392.
Closes #396.

Part of #391. This PR remains draft until the pending integration and acceptance work below is complete.

Remaining acceptance includes production adoption/storage guides, standalone release and Docker UID evidence, unreadable/unusable mount cases, retained-handle lock evidence, and the accepted-#275 re-audit.

Test plan

  • cargo test --workspace --all-targets
  • cargo clippy --workspace --all-targets --all-features -- -D warnings
  • cargo fmt --all -- --check
  • cargo check --workspace --all-targets --features "fastly cloudflare spin"
  • Full WASM adapter builds for Fastly, Spin, and Cloudflare
  • examples/app-demo workspace: cd examples/app-demo && cargo test --workspace --all-targets
  • Docs build: cd docs && npm run lint && npm run format && npm run build
  • Manual testing via edgezero serve --adapter axum
  • cargo test -p edgezero-core --all-targets
  • cargo test -p edgezero-adapter-axum --all-targets --all-features, including all 13 production-hosting subprocess tests
  • cargo check -p edgezero-adapter-spin --target wasm32-wasip2 --features spin
  • cargo check -p edgezero-core --test production_handlers --target <target> for wasm32-wasip1, wasm32-wasip2, and wasm32-unknown-unknown
  • git diff --check

Checks passed on this branch, not on a merge with the newer prerequisite. Native subprocess tests use debug fixtures and the current user's UID, not a release container image. Shutdown cancellation is cooperative; blocking application code still requires a supervisor hard-kill boundary.

Checklist

  • Changes follow CLAUDE.md conventions
  • No Tokio deps added to core or adapter crates
  • Route params use {id} syntax, not :id
  • Types imported from edgezero_core, not http
  • Store wiring goes through KvRegistry / ConfigRegistry / SecretRegistry
  • New code has tests
  • No secrets or credentials committed

TS cutover follow-up, November 11

Adds startup-only router installation without rebuilding application clocks, platform metadata, or admission/egress policies. This lets a prepared-store initializer assemble application routes before the listener is shared.

Current head: 1b5e361e4896ad098fc26925556eb58b52ecdbb6.

Local combined-RC qualification at 36f8e9c9c2c93661aaa73f2ca956cec3bc9e4c92: workspace tests 2,947 passed / 6 ignored; all-feature Axum 568 passed, including 36 production-hosting, 6 AWS startup and 4 celld startup cases; core doctests 19 passed / 13 ignored; strict workspace Clippy, formatting, feature compilation and Spin WASM check passed under Rust 1.98.1. Narrow TS consumer proof under Node 24.20.0: all four real proxy tests and typed prepared config/secrets/router startup passed. This is not final TS-stack/image/deployment qualification. Fresh hosted checks are still required.

Owned follow-up review checkpoint

@ChristianPavilonis
ChristianPavilonis force-pushed the issue-392-396-production-hosting-spec branch from 3ab2209 to 49ba021 Compare November 10, 2026 19:40

@prk-Jr prk-Jr left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review

Summary

Follow-up review at 6d1dda16. The blocking findings from the two earlier reviews (at 9f090e57) are still open. The commits since then add App::set_router, carry received-head preparation through, merge #275 repairs and restore a Fastly fixture. None of them addresses those findings, and none of the inline threads has a reply yet. So this stays at changes requested.

Still open (not repeated inline)

  • The subprocess suite (production_hosting.rs) needs axum + test-utils, and no test.yml step enables both, so CI compiles it and runs 0 tests.
  • Embedded runs install a global logger, so a second run_app_with_options* call in one process fails. Standalone startup installs the logger after App::build, and an App::build failure loses the startup …: prefix.
  • Dev startup no longer logs the listening URL, and every bind error reads address unavailable.
  • Dev and embedding Ctrl-C/SIGTERM behaviour changed: a fixed 10 s drain, and serve() now takes over SIGTERM. docs/guide/adapters/axum.md hasn't been updated.
  • Closes #392 / Closes #396 while acceptance work is still listed as remaining.

New findings

Non-blocking

  • 🤔 App::set_router is new public core API added after review, and isn't documented (app.rs:592, see inline)
  • ⛏ PR body is out of date and contradicts itself. The top says "Ready for review" at publication head 9f090e57. A later section says the current head is 1b5e361e, and the Summary still says it "remains draft". The reviewed head is 6d1dda16. The Changes table doesn't list App::set_router.

Note on an earlier finding

  • The concern that the [[bin]] edgezero-hosting-fixture target ships in the published crate has little impact. The workspace sets publish = false and edgezero-adapter-axum inherits it, and the bin needs test-utils to build. It's still fine to move it if you want a cleaner manifest.

CI Status (head 6d1dda16)

  • Hosted: 27 checks passed and 2 failed (cache-restore-smoke, cache-new-revision-smoke in Deploy actions). Both fail on a missing Actions source cache (SOURCE_CACHE_HIT: false). The same jobs fail on #407, and the cache seed job fails on #275. This PR doesn't touch that action, so it looks like a CI-environment issue, not a code defect.
  • Local (Rust 1.98.1, macOS): cargo test -p edgezero-adapter-axum --all-features --test production_hosting passed 13/13 in two runs with tests in parallel. That doesn't rule out the parallel port race reported earlier; two runs is too few to settle it.
  • Local: edgezero-core received_head_preparation / production_handlers tests and the startup_state lib test passed.
  • Not re-run locally: the full workspace suite, clippy, WASM builds and examples/app-demo.

/// Previous router dispatch tokens no longer belong to this App. This is not live reload;
/// finish initialization before sharing the App with a listener.
#[inline]
pub fn set_router(&mut self, router: RouterService) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 New public core API added after review, and not documented

set_router replaces the whole RouterService after Hooks::configure has run. That throws away anything attached to the previous router, including state added with insert_state, middleware and the received_head_preparer, while the admission and egress policies configured against the old routes stay in place. That may be what the TS cutover needs, but:

  • The design doc and the PR's Changes table don't mention this API.
  • prepared_startup_router_keeps_application_clock_and_platform checks clock, platform and state, but not the promise in the doc comment that previous dispatch tokens stop working. From the source the promise holds: dispatch_resolved checks Arc::ptr_eq(&self.inner, &resolved.owner), and old tokens keep the old router alive. A test would lock that in.

Fix: add one rustdoc line saying that router-attached state, middleware and the head preparer are replaced, add an assertion that an admission token issued before set_router gets a 500 afterwards (as startup_state_forks_router_and_invalidates_old_admission does for insert_state), and list the method in the PR body. Because this serves TS work, it could also move to its own PR, in line with the earlier out-of-scope note.

@aram356

aram356 commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator

CREATOR: From the server-side identity plan (IABTechLab/trusted-server#1559, c7dc9de77): Take run_production_app_with_options_and_initializer from 4ea99467; the Trusted Server RC's native startup calls it.

Part of #476

@ChristianPavilonis
ChristianPavilonis removed this pull request from stack #465 November 13, 2026 16:48
@ChristianPavilonis
ChristianPavilonis changed the base branch from docs/outbound-http-spec to review/pr275-cutover-followup-20261112 November 13, 2026 16:48
@ChristianPavilonis
ChristianPavilonis removed this pull request from stack #500 November 13, 2026 17:32
@ChristianPavilonis
ChristianPavilonis changed the base branch from review/pr275-cutover-followup-20261112 to issue-459-outbound-compat November 13, 2026 17:32
aram356 added a commit that referenced this pull request Nov 14, 2026
Brings #275 at e9ffaa8 and the refreshed heads of #499, #460, #406,
#407, #409, #414, #422 and #439. Each PR merged the one below it, so
this merge has no conflicts.

Two edits fit #469 to #275, as #469's open review asks:

- #469's ingress JA4 test calls into_core_request with one argument.
  #275 now passes the Axum outbound executor through that function, so
  the test gets an executor handle the way the other ingress tests do.
- The Axum guide names EdgeZeroAxumService, which #275 removes and
  check_outbound_legacy_api.sh rejects. The guide now names only
  AxumDevServer::with_trusted_ja4_header.
aram356 added a commit that referenced this pull request Nov 16, 2026
Brings #275 at b65e714 and the refreshed heads of #499, #460, #406,
#407, #409, #414, #422 and #439. Each PR merged the one below it, so
this merge has no conflicts. #409 resolved the one conflict on the way
up, in edgezero-core's app.rs imports, by keeping both.
aram356 added a commit that referenced this pull request Nov 17, 2026
Brings #275 at ea2549f and the refreshed heads of #499, #460, #406,
#407, #409, #414, #422 and #439, including #422's stock-celld store
service and its merge into #439. Each PR merged the one below it, with
no conflicts on the way up, so this merge has no conflicts.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Axum: make production configuration and local storage explicit Axum: define a production server lifecycle with bounded graceful shutdown

4 participants